Method and apparatus for compaction of data received over a network

ABSTRACT

Methods, apparatuses, and storage media associated with compaction of data from one or more computing devices are disclosed. In various embodiments, one or more Internet of Things (IoT) devices may transmit information to a computing system. The computing system may group together raw data received from these one or more IoT devices based on a shared attribute. The computing system may select a compaction scheme to represent the knowledge conveyed by a group of the raw data. The computing system may apply this compaction scheme to the group of raw data to generate data that is representative of the group of raw data. Other embodiments may be disclosed or claimed.

RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.14/911,223 filed Feb. 9, 2016, entitled “METHOD AND APPARATUS FORCOMPACTION OF DATA RECEIVED OVER A NETWORK”, now U.S. Pat. No.10,015,272 issued Jul. 3, 2018 which is a national phase entry under 35U.S.C. § 371 of International Application No. PCT/CN2015/074093, filedMar. 12, 2015, entitled “METHOD AND APPARATUS FOR COMPACTION OF DATARECEIVED OVER A NETWORK”, which designated, among the various States,the United States of America. The contents of PCT/CN2015/074093 and U.S.Ser. No. 14/911,223 are hereby fully incorporated by reference in theirentireties.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart by inclusion in this section.

Data from various computing devices may stream at unprecedented rates.As millions of devices may send data many times (e.g., at rates ofseveral times each second, minute, etc.), the volume of data that mustbe processed, indexed, and stored raises severe economic and technicalhurdles in provisioning storage capacity and bandwidth.

Compression may be one approach to managing such voluminous data. Incompression, original data may be stored using a reduced number of bits.Compression may be lossy or lossless. In lossy compression, the numberof bits required to store original data may be reduced by discarding“unimportant” data. In lossless compression, original data may becompressed by eliminating statistically redundant data so that data maybe decompressed to exactly its original value. However, compressionstill requires appreciable storage capacity and so alternativeapproaches to data storage reduction may be examined.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be described by way ofexemplary embodiments, but not limitations, illustrated in theaccompanying drawings in which like references denote similar elements,and in which:

FIG. 1 is a block diagram illustrating an example environment thatincludes a system for compacting data received from a plurality ofdevices, in accordance with various embodiments.

FIG. 2 is a block diagram illustrating a data compaction system thatincludes modules for data compaction, data uncompaction, and queryprocessing, in accordance with various embodiments

FIG. 3 is a block diagram illustrating an example compaction table forstoring representative data, in accordance with various embodiments.

FIG. 4 is a block diagram illustrating example operations associatedwith compaction of data from a plurality of devices, in accordance withvarious embodiments.

FIG. 5 is a block diagram illustrating an example of generation ofrepresentative data, in accordance with various embodiments.

FIG. 6 is a block diagram illustrating another example of generation ofrepresentative data, in accordance with various embodiments.

FIG. 7 is a graph illustrating a set of readings from at least oneInternet of Things device, in accordance with various embodiments.

FIG. 8 is a graph illustrating a linear function fit to a set ofreadings from at least one Internet of Things device, in accordance withvarious embodiments.

FIG. 9 is a graph illustrating a quadratic function fit to a set ofreadings from at least one Internet of Things device, in accordance withvarious embodiments.

FIG. 10 is a graph illustrating a piecewise fit to a set of readingsfrom at least one Internet of Things device, in accordance with variousembodiments.

FIG. 11 is a graph illustrating an example representation of valuesbased on respective relationships to a principal value, in accordancewith various embodiments.

FIG. 12 is a flow diagram illustrating a method for compacting data fromone or more Internet of Things devices, in accordance with variousembodiments.

DETAILED DESCRIPTION

Various aspects of the illustrative embodiments will be described usingterms commonly employed by those skilled in the art to convey thesubstance of their work to others skilled in the art. However, it willbe apparent to those skilled in the art that alternate embodiments maybe practiced with only some of the described aspects. For purposes ofexplanation, specific numbers, materials, and configurations are setforth in order to provide a thorough understanding of the illustrativeembodiments. However, it will be apparent to one skilled in the art thatalternate embodiments may be practiced without the specific details. Inother instances, well-known features are omitted or simplified in ordernot to obscure the illustrative embodiments.

Various operations will be described as multiple discrete operations, inturn, in a manner that is most helpful in understanding the illustrativeembodiments; however, the order of description should not be construedas to imply that these operations are necessarily order dependent. Inparticular, these operations need not be performed in the order ofpresentation. Further, descriptions of operations as separate operationsshould not be construed as requiring that the operations be necessarilyperformed independently and/or by separate entities.

Descriptions of entities and/or modules as separate modules shouldlikewise not be construed as requiring that the modules be separateand/or perform separate operations. In various embodiments, illustratedand/or described operations, entities, data, and/or modules may bemerged, broken into further sub-parts, and/or omitted.

The phrase “in one embodiment” or “in an embodiment” is used repeatedly.The phrase generally does not refer to the same embodiment; however, itmay. The terms “comprising,” “having,” and “including” are synonymous,unless the context dictates otherwise. The phrase “A/B” means “A or B.”The phrase “A and/or B” means “(A), (B), or (A and B).” The phrase “atleast one of A, B and C” means “(A), (B), (C), (A and B), (A and C), (Band C) or (A, B and C).”

FIG. 1 illustrates an example environment 100 that includes a system 110for compacting data received from a plurality of devices 102 a-b,according to teachings of the present disclosure. The environment 100may include, but is not limited to, one or more devices 102 a-b, anexternal computing system 150, and a data compaction system 110, whichmay incorporate the teachings of the present disclosure. The devices 102a-b and the external computing system 150 may be communicatively coupledwith the data compaction system 110 at least as shown, for example, overone or more wired and/or wireless networks, including the Internet.

In various embodiments, each of the devices 102 a-b may be an Internetof Things (IoT) device. Examples of IoT devices include phones,smartphones, medical gadgets, meters or gauges (e.g., energy,temperature, humidity, pulse rate, pressure, voltage, current gauges,etc.), appliances, vehicles (e.g., Internet of Vehicles (IoV) devices),or essentially any other device configured with an embedded computingdevice operable to transmit data over a network (e.g., the Internet). Insome embodiments, IoT devices may include cooperative mobile devices,such as devices configured to share data, resources, and/or processingfunctionality.

In embodiments, the devices 102 a-b may be configured to capture rawdata comprising a plurality of values. The devices 102 a-b may lack diskstorage and may only include respective NAND or NOR flash of relativelysmall capacity that may be reserved for memory. Therefore, datacompaction may efficiently represent raw data captured by the devices102 a-b. The raw data 104 a-b provided by the devices 102 a-b may beexpected to exhibit a variety of characteristics that may be leveragedfor efficient data representation: (1) bounded deviation, (2) thresholdsensitivity, and (3) skew.

With respect to bounded deviation, changes in values comprising the rawdata 104 a-b from the devices 102 a-b are frequently limited inmagnitude over small time periods. That is, relatively large changes invalues read by one of the devices 102 a-b may occur occasionally, butrelatively small changes in values in relatively small time periods areto be expected. The devices 102 a-b may be configured to adapt and/or tocompensate for various readings over relatively short time periods. Forexample, methane (CH₄) and/or carbon dioxide (CO₂) emission rates may beread by one of the devices 102 a-b and oscillations in values may bedetected, for example, in a diurnal cycle. Similarly, one of the devices102 a-b may be configured to detect temperature, and the detectedtemperature may gradually vary second by second.

With respect to threshold sensitivity, a signal detected by one of thedevices 102 a-b may exceed a threshold value in either directionrelative to a central value in order to convey information. Inembodiments, the devices 102 a-b may include respective sensitivitythresholds. For example, one of the devices 102 a-b may be a cameraconfigured to detect traffic. The ability of the one of the devices 102a-b to capture license plates, driver faces, etc. (e.g., resolution),may not be germane for traffic analytics; rather, a relative change intraffic congestion may be useful for traffic analytics (e.g., raw data104 a-b may convey whether traffic has increased by a thresholdpercentage over a certain duration).

With respect to skew, the devices 102 a-b may convey raw data 104 a-bthat indicates mass around one or more bands close to a central value,while other values conveyed by the raw data 104 a-b in bands fartheraway from the central value may be relatively infrequent. For example, acommonly occurring Normal distribution may have 68.2% mass in a ±σ bandand 95.4% mass in the ±2σ about the mean. The prevalence of skew indeviations indicates that small numbers of quantizations may besufficient to capture critical gradations in data.

The concepts of bounded deviation, threshold sensitivity, and/or skewmay be leveraged for efficient data representation. In embodiments, oneor more compaction schemes may be based on one or more of boundeddeviation, threshold sensitivity, and/or skew. For example, skew incombination with threshold sensitivity may be conducive to a compactionscheme comprising a coarse-grained bitmap that is sufficient to detectthreshold value crossings.

In embodiments, each of the devices 102 a-b may transmit raw data 104a-b to the data compaction system 110. For example, the devices 102 a-bmay stream the raw data 104 a-b to the data compaction system 110 at arelatively high rate (e.g., many times each second or minute).

Except for the teachings of the present disclosure, the data compactionsystem 110 may be an edge device. Examples of edge devices includerouters, routing switches, integrated access devices (IADs),multiplexers, and so forth. In another embodiment, the data compactionsystem 110 may be communicatively coupled with one or more edge devicesto receive raw data 104 a-b from the devices 102 a-b. In suchembodiments, the data compaction system 110 may be included in a server,a desktop computer, a laptop computer, or essentially any other computerdevice adapted to transmit signals over a network.

According to embodiments, the data compaction system 110 may beconfigured to apply at least one compaction scheme to the raw data 104a-b to represent that raw data 104 a-b. Compaction of the raw data 104a-b may allow an edge device to retain more data without communicationwith a block device (e.g., a storage device, such as a hard disk drive,CD-ROM drive, and/or flash drive).

In various embodiments, the data compaction system 110 may comprisecircuitry and/or logic to group values included in the raw data 104 a-bbased on at least one attribute associated with at least one of thedevices 102 a-b. The data compaction system 110 may select at least onecompaction scheme based on the values of the group and generaterepresentative data based on application of the selected at least onecompaction scheme to the values of the group. Accordingly, the datacompaction system 110 may select and apply domain-appropriate compactionschemes. Further, the data compaction system 110 may adapt to domainand/or raw data 104 a-b, for example, through dynamic selection of acompaction scheme. The data compaction system 110 may be configured tostore the representative data, for example, in a table of a databaseincluded in or communicatively coupled with the data compaction system110.

The environment 100 may further include an external computing system150. According to embodiments, the external computing system 150 may bea server, a desktop computer, a laptop computer, or essentially anyother computer device adapted to transmit signals over a network. Theexternal computing system 150 may be configured to transmit one or morequeries 140 to the data compaction system 110. In various embodiments,the one or more queries 140 may be, for example, database queries.

In response to the one or more queries 140, the data compaction system110 may be configured to identify responsive data to transmit to theexternal computing system 150 as results 142. In some embodiments, thedata compaction system 110 may identify raw data from raw data 104 a-bas responsive data and transmit the same to the external computingsystem 150 as results 142.

In some embodiments, the data compaction system 110 may be configured togenerate the responsive data based on the representative data. Accordingto various embodiments, the representative data may include anindication of how to generate the responsive data, such as a functionand/or a bitmap index. Based on the query, the data compaction system110 may identify representative data and generate responsive data basedon the representative data to transmit to the external computing system150 as results 142. For example, the data compaction system 110 maygenerate responsive data through a randomized algorithm forinterpolation based on the representative data and/or application of afunction indicated by the representative data.

Turning now to FIG. 2, a block diagram illustrates a data compactionsystem 200 that includes modules 206, 208, 210 for data compaction, datauncompaction, and query processing, in accordance with variousembodiments. The data compaction system 200 may be an embodiment of thedata compaction system 110, as illustrated in FIG. 1.

As shown, for the illustrated embodiments, the data compaction system200 may include one or more processors 202, memory 204, and a networkinterface 230, coupled with each other at least as shown. The processor202 may be configured to execute instructions of a compaction module206, an uncompaction module 208, and a query processing module 210,loaded from memory 204. The compaction module 206, uncompaction module208, and query processing module 210 may be implemented in software,hardware, firmware, or a combination thereof.

The processor 202 is intended to represent a broad range of processors,such as single or multi-core processors of various execution speeds andpower consumptions. Similarly, the memory 204 is intended to representmemory of various architectures with one or more levels of caches, andof various types, such as dynamic random access, flash, and so forth. Insome embodiments, the processor 202 and the memory 204 (as well asadditional components) may be integrated, such as in a system on chip(SoC).

The network interface 230 may be comprised of transmit circuitry and/orreceive circuitry. The network interface 230 may be configured totransmit and/or receive data over any number of different wired and/orwireless networks. Accordingly, the one or more network(s) 234 is (are)intended to represent a broad range of networks known in the art.Examples of network(s) 234 may include wired or wireless, local or widearea, private or public networks, including the Internet.

The data compaction system 200 may include or may be communicativelycoupled with (e.g., over the network(s) 234) storage 240. The storage240 is intended to represent any storage media suitable to storecompacted data. Examples of the storage 240 include flash (e.g., NANDflash), electrically erasable programmable read-only memory (EEPROM),hard disk drive (HDD), and so forth. In some embodiments, the storage240 may comprise a database. The storage 240 may include a plurality ofcompaction tables 242, for example, as tables in a database. The storage240 may be configured to store both raw and compacted data.

In various embodiments, the network interface 230 may be configured toreceive, over the network(s) 234, and process raw data from one or moreIoT devices. The compaction module 206 may be provided this raw data bythe network interface 230. The compaction module 206 may be configuredto identify at least one attribute common to a plurality of valuescomprising the raw data and associated with the one or more of the IoTdevices from which the raw data is received. For example, the compactionmodule 206 may be configured to detect an identifier associated with anIoT device (e.g., a device ID) so that the compaction module 206 maygroup together values from a same IoT device.

According to embodiments, the compaction module 206 may be configured todetermine an amount of values comprising raw data to be groupedtogether. A first plurality of aggregated values may be within adiscrete range and, therefore, may be conducive to compaction using afirst compaction scheme, whereas another plurality of values may bedisparate and, therefore, would be better suited by another compactionscheme.

In embodiments, the compaction module 206 may be configured to determinea compaction scheme that is to represent the first plurality of values.A compaction scheme may be any approach to attempting to reduce thestorage consumed by the raw data through reduction of the actualinformation theoretic content without affecting the meaningful use ofdata.

Examples of compaction schemes include function fitting, piecewisefitting, relation to a principal value, and bitmap index compaction. Ina function fitting compaction scheme, the compaction module 206 may beconfigured to construct a function (e.g., a linear or quadraticfunction) to aggregated values. Similarly, for a piecewise compactionscheme, the compaction module 206 may be configured to fit a piecewiselinear function to aggregated values or a plurality of groups ofaggregated values.

With respect to a compaction scheme based on relation to a principalvalue, the compaction module 206 may represent values based on theirrespective relationships to a principal value. For example, thecompaction module 206 may select a principal value that is the mean of agroup of aggregated values and each aggregated value may be representedby its percentage deviation from the principal value. In anotherembodiment, the compaction module 206 may observe that aggregated valuesare clustered around a principal value, such as a mean μ. Based onaggregated values, the compaction module 206 may identify a standarddeviation σ. Accordingly, aggregated values may be represented by anumber of values within a standard deviation σ (or a multiple thereof)from a mean μ.

With respect to a compaction scheme based on a bitmap index, thecompaction module 206 may be configured to populate one or more bitmapcolumns or rows based on aggregated values. For example, there may beeight possible bands over which an attribute associated with a group maybe distributed. In one embodiment, an eight-column bitmap index may bepopulated based on the values to indicate aggregated values of a group.In another embodiment, e.g., if the skew is high, then a one-columnbitmap index may be used to represent the aggregated values to indicatewhether each value occurs in a high density band (an additional columnmay be populated to indicate whether an attribute occurs at all).

The compaction module 206 may be configured to dynamically select acompaction scheme in response to received raw data. In variousembodiments, the compaction module 206 may be configured to aggregatevalues comprising the raw data and may be configured to select variouscompaction schemes for different aggregations of values. For example,the compaction module 206 may be configured to select a first compactionscheme for a long, tranquil section of raw data from an IoT device, butswitch to a second compaction scheme for a jittery section of raw datafrom the same IoT device, and finally return to the first compactionscheme for a third section of raw data. In another example, thecompaction module 206 may apply a function fitting compaction scheme toone or more sections of values of a group, but may adaptively blend indeviation encoding to module precision in data fitting.

The compaction module 206 may be configured to apply at least oneselected compaction scheme to a plurality of values, such as aggregatedvalues of a group. Application of a selected compaction scheme to aplurality of values may cause the compaction module 206 to generaterepresentative data. The representative data generated by the compactionmodule 206 may vary according to the selected compaction scheme. Forexample, the representative data may be a function, a bitmap index, aprincipal value, and so forth. The representative data may furtherinclude an indication of a number of values comprising raw datarepresented by the representative data. In embodiments, the compactionmodule 206 may further generate the representative data to include othervalues, such as a minimum value, maximum value, and so forth.

The compaction module 206 may be configured to store representative datain the storage 240, for example, in at least one compaction table 242.Because the compaction module 206 may be configured to apply varyingcompaction schemes to different values of a group, the compaction module206 may be configured to store different representative data indifferent columns of one or more compaction tables 242 or even differentrepresentative data in a same column of one of the compaction tables242.

In some embodiments, the compaction module 206 may be configured tostore raw data in the storage 240. For example, if the compaction module206 is unable to select a compaction scheme and/or determines thatcertain raw data is unsuitable for compaction, the compaction module 206may be configured to store that raw data in the storage 240. However, inone embodiment, the compaction module 206 may be configured to discardraw data, for example, after representative data is generated toindicate that raw data.

The compaction module 206 may be further configured to apply datacompression to one or both of raw data and/or representative data. Forexample, the compaction module 206 may be configured to apply bitmapcompression (e.g., word-aligned hybrid) to a bitmap index ofrepresentative data.

In various embodiments, the network interface 230 may be configured toreceive, over the network(s) 234, and process one or more queries froman external computing system. In various embodiments, the query maycomprise a request for data associated with one or more IoT devices. Forexample, a query may indicate an attribute according to which raw datamay have been grouped (e.g., an attribute associated with representativedata). In various embodiments, the query may comprise a database query.

The query processing module 210 may be configured to receive one or morequeries from the network interface 230. In various embodiments, thequery processing module 210 may comprise a database management system(DBMS). Examples of DBMSs include Oracle of Oracle Corporation,headquartered in Santa Clara, Calif., Postgres of PostgreSQL GlobalDevelopment Group, and Hbase of the Apache Software Foundation,headquartered in Forest Hill, Md. However, the query processing module210 may be configured for use based on relatively stringent requirementsof the processor 202, memory 204, and/or storage 240, for example, wherethe data compaction system 200 is included in an edge device withlimited processor and/or memory bandwidth.

In some embodiments, the query processing module 210 may be configuredto access the storage 240 based on a query. For example, the queryprocessing module 210 may be configured to respond to a query with rawdata. Thus, the query processing module 210 may be configured to respondto a “retrieve all” query so that all data is returned by the storage240.

In various embodiments, the query processing module 210 may beconfigured to interact with the uncompaction module 208, for example, toretrieve data with which to respond to a query. The uncompaction module208 may be configured to generate data (e.g., responsive data) based onrepresentative data stored in one of the compaction tables 242. Forexample, the uncompaction module 208 may be configured to generate a setof values based on representative data stored in one of the compactiontables 242, and the representative data may indicate the number ofvalues to be generated. In embodiments, the uncompaction module 208 maynot exactly duplicate values of raw data; however, the uncompactionmodule 208 may un-compact the representative data so that salientinformation is generated, for example, for analytics.

In one embodiment, the uncompaction module 208 may be configured togenerate values based on a function. For example, representative datamay comprise a function and a number of values for which that functionis valid. According, the uncompaction module 208 may be configured togenerate the number of values using the function and provide thosevalues to the query processing module 210 as data response to a query.Similarly, for a piecewise compaction scheme, the uncompaction module208 may be configured to generate a plurality of values using apiecewise linear function.

With respect to a compaction scheme based on relation to a principalvalue, the uncompaction module 208 may generate values based on theirrespective relationships to a principal value. For example, therepresentative data may include a principal value, an indication of anumber of values represented by the representative data, and respectivedeviations for each of the values from the principal value. Accordingly,the uncompaction module 208 may calculate each value of the number ofvalues based on each deviation from the principal value.

In another embodiment, the uncompaction module 208 may be configured tointerpolate values based on the representative data. For example, theuncompaction module 208 may be configured to generate values based on arandomization function that constrains each generated random value towithin a deviation (or a multiple thereof) from the principal value.

With respect to representative data comprising a bitmap index, theuncompaction module 208 may be configured to generate values based oncolumns or rows based on aggregated values. For example, theuncompaction module 208 may be configured to generate values in one ormore bands based on one or more bitmaps that indicate whether suchvalues occurred in the one or more bands in raw data.

The uncompaction module 208 may be configured to provide data based onun-compaction of representative data (e.g., generated data) to the queryprocessing module 210. Accordingly, the query processing module 210 maybe configured to respond to one or more queries with that data providedby the uncompaction module 208.

Now with reference to FIG. 3, a block diagram illustrates an examplecompaction table for storing representative data, in accordance withvarious embodiments of the present disclosure. In relation toembodiments described in FIG. 2, the compaction table 300 may be anembodiment of one of the compaction tables 242 and the compaction module206 may be configured to populate data in the compaction table 300.

In embodiments, the compaction table 300 may be comprised of a pluralityof structures: metadata 304, index data 306, and record data 308. Themetadata 304 may describe how the compaction table 300 is stored and/ororganized. The metadata 304 may include a metadata structure (or map)that includes information describing policies, attributes, and/orproperties of the listed attributes.

In some embodiments, the metadata 304 may describe a key generationpolicy, an aggregation policy, a data ordering policy, group attributesand their properties, and compacted attributes and their properties. Thekey generation policy may describe how to automatically generate newunique keys for a new record that is a compaction of data from raw data(e.g., representative data). The key generation policy may specify anamespace and a mapping of keys between raw data and representativedata.

The aggregation policy of the metadata 304 may describe the criteria bywhich raw data is compacted and/or stored as raw data. For example, asimple criterion may include a policy to compact sixteen consecutiverecords of raw data for a first attribute or compact all records of rawdata for another attribute that spans one minute. More complex criteriamay include conditions governing loss of precision, magnitude ofdeviation from compacted representation, and so forth. The data orderingpolicy may describe how records of raw data in a same group are orderedbefore compaction—e.g., by timestamp, original key values, and so forth.

The metadata 304 may further identify group attributes and properties. Agroup attribute may identify an attribute in raw data on whichcompaction is performed. Properties of a group attribute may includewhat data type it is and how it may be transformed into a new attributethrough compaction—e.g., individual absolute magnitude values of rawdata may be transformed into deviations from a principal value. Inembodiments, compaction of values may be independent across differentattributes and, therefore, there may be a plurality of group attributes.

For each group attribute, the metadata 304 may identify an aggregationattribute associated with compacted data (e.g., representative data)that carries the transformed value. Properties of an aggregationattribute may include the type of attribute before and after compaction,a compaction scheme and an un-compaction scheme, and parameters forrespective schemes. A compaction scheme may be explicitly indicated(e.g., “linear function fitting”), or may be specified by a set ofparameterized rules (e.g., an indication of how a compaction module isto select a compaction scheme, such as an indication that a compactionscheme is to be selected only if preserves accuracy of the raw data togreater than or equal to 0.975). In embodiments of parameterized rules,a compaction module may decide, based on raw data, which compactionscheme yields acceptable results and apply it for compaction.

In embodiments, the index data 306 may maintain various indexing systemsadapted for compacted data so that suitable data is un-compacted, forexample, in response to one or more queries. Because compaction reducesthe volume of data both in bit size and number of stored values, indicescan be reduced in breadth but carry more depth. Aggregation attributes(stored as part of record data 308) effectively may be a type of indexsince they summarize data over a span and therefore allow an entire spanto be skipped if summaries show no intersection with the parameters of aquery. The index data 306 may include indices such as trees, tries,hashes, table index, as well as bitmap indices.

The record data 308 may store compacted data (e.g., representativedata). Record data 308 may further include raw data, for example, thatwas not compacted due to an unsuitable match with existing compactionschemes. Record data 308 may further comprise other summarization data,such as statistics gathered over raw data (e.g., minimum value, maximumvalue, cardinalities, variances, etc.) that may be employed forun-compaction and/or query processing. For example, when scope filtering(e.g., include values between a minimum and a maximum over a range) ispossible over an aggregation attribute, statistical information mayfacilitate filtering without un-compaction.

Turning to FIG. 4, a block diagram illustrates operations associatedwith compaction of data from a plurality of IoT devices 420, inaccordance with various embodiments. In relation to FIG. 2, thecompaction module 400 may be an embodiment of the compaction module 206and the table data 416 may be an embodiment of the compaction tables242.

In embodiments, as raw data is received from the plurality of IoTdevices 420, the data grouping module 402 may be configured to grouprecords (e.g., values) of raw data according to group attributes 424 asindicated in the table metadata 412. The data grouping module 402 maystore these grouped original records 422 of the raw data in storage fortransient group records 414.

Thereafter, an aggregate monitor module 404 may receive an aggregationpolicy 428 from table metadata 412 and use the aggregation policy 428 toselect and retrieve qualified groups 434 from the transient grouprecords 414. The aggregate monitor module 404 may then provide thequalified groups 434 to a record aggregator 406 as grouped records 436.The record aggregator 406 may apply aggregate attributes 426 from thetable metadata 412 to build aggregated records, such as throughapplication of a compaction scheme to generate representative data.Further, the record aggregator 406 may use a key generation policy fromthe table metadata 412 to associate the aggregated records with newkeys. Finally, the record aggregator 406 may store aggregated records inconnection with key association in the table data 416 as representativedata 432.

With respect to FIG. 5, a block diagram illustrates an example ofgeneration of representative data, in accordance with variousembodiments. In embodiments, original records 502 comprising raw datamay include three fields: a key field (M_ID), a group attribute field(G), and an attribute value (A). Similarly, an aggregated record 504comprising representative data may include three fields: a new key of anaggregated record comprising (A_ID), the group attribute (G), and thecompacted attribute value (A′).

In the original records 502 comprising raw data, all original records502 having the same group attribute g1 are grouped together. In theillustrated embodiment, the sixty-four records 502 comprising raw datainclude ten unique attribute values a1-a10. A compaction scheme 506 maybe applied to the grouped original records 502 to represent the uniquevalues a1-a10 of the sixty-four original records 502. Application of thecompaction scheme may cause an aggregated value aggr(a1-a10) to begenerated. This aggregated value may be stored in the aggregated record504 comprising representative data, in connection with the new key901210 and the group attribute g1.

Similarly, FIG. 6 depicts a block diagram illustrating another exampleof generation of representative data based on raw data, in accordancewith various embodiments. As illustrated, original records 602comprising raw data from a buoy sensor may include three fields: adevice identification field (Device_id), a time field (Time and informat of yyyymmddhhmmss), and value (Wave_height). In the illustratedembodiment, the original records 602 may be grouped according to asingle device ID 1, and twenty-five records 602 may be processed over aone minute interval.

In embodiments, a compaction scheme 610 may be applied to the originalrecords 602. Application of the compaction scheme 610 may be based onmetadata 604 that may describe how the compaction scheme 610 is to beapplied and/or how representative data is to be generated therefromand/or stored. In the illustrated embodiment, the metadata 604 mayinclude a key generation policy describing how to generate a keyassociated with representative data based on the device ID and the timeof the original records 602, an aggregation policy describing a numberof records to which the compaction scheme 610 is to be applied, anordering policy describing how the records are to be ordered (e.g.,sequentially by time), a group attribute describing an attribute overwhich the original records are grouped and whether that attribute isaffected during application of the compaction scheme 610, and otheraggregation attributes describing the type of compaction scheme to beapplied (e.g., piecewise fitting) and what field the compaction schemeis to be applied (e.g., wave height).

Based on application of the compaction scheme 610, an aggregated record608 comprising representative data is generated. This aggregated record608 may include a device ID field indicating an identification of thebuoy sense represented in the aggregated record 608, a time fieldindicating a time represented in the aggregated record 608 and a numberof original records 602 comprising raw data represented in theaggregated record 608, and a field for compacted data. Because thecompaction scheme is applied to the wave height field of the originalrecords 602, the field for compacted data is also wave height. In thisfield, piecewise fitting through the compaction scheme 610 has generatedtwo functions: one function representing ten of the original records 602and another function representing fifteen of the original records 602.Other statistical data may be represented in this field, such as aminimum value, a maximum value, and/or a number of values that may begenerated based on the representative data.

In reference to FIGS. 7-10, a plurality of graphs illustrate results ofdifferent applications of compaction schemes to original records, inaccordance with various embodiments. In FIG. 7, a graph illustratesreadings 702 overs a three-second interval from a power meter on thevertical axis. Readings 702 from the power meter are illustrated asthirty-two dots. The readings 702 indicate a slightly rising trend overthe three-second interval, with some amount of oscillation around thattrend.

In FIG. 8, the same data as FIG. 7 is illustrated but with a lineartrend line 804 superimposed. In the illustrated embodiment, the lineartrend line 804 may be the function y=0.041x+24.97. Accordingly,capturing just two parameters may provide a compact approach torepresenting thirty-two readings 702. The loss of precision throughrepresentation of the readings 702 with a function represented by thelinear trend line 804 may be sufficient to convey the informationindicated by the readings 702.

Similarly, in FIG. 9, the same data as FIG. 7 is illustrated but with amore snug quadratic fit. In the illustrated embodiment, the quadraticline 906 may be the function y=−0.0029x²+0.136x+24.437. Accordingly,capturing three parameters may provide a compact approach torepresenting thirty-two readings 702. The loss of precision throughrepresentation of the readings 702 with a function represented by thequadratic line 906 may be sufficient to convey the information indicatedby the readings 702. The storage savings provided through compaction maycompound dramatically as the dimensionality of data increases.

Similarly, in FIG. 10, the same data as FIG. 7 is illustrated but withpiecewise linear fitting. In the illustrated embodiment, a first linearfunction line 1008 may be the function y=0.0632x+24.437, a second linearfunction line 1010 may be the function y=0.0803x+25.47, and a thirdlinear function line 1012 may be the function 0.0018x+25.91.Accordingly, capturing six parameters across three linear functions mayprovide a compact approach to representing thirty-two readings 702. Theloss of precision through representation of the readings 702 with afunction represented by the piecewise linear functions may be sufficientto convey the information indicated by the readings 702. In otherembodiments, a piecewise fit may be negative in some instances toachieve a closer fit with data. Further, intervals of a piecewise fitmay be of variable length in order to achieve both a close fit and datareduction.

In reference to FIG. 11, a graph illustrates application of compactionschemes based on association of values comprising raw data to aprincipal value, in accordance with various embodiments. In someembodiments, a function may not fit data snugly, such as when readingsoscillate around a mean value. In the embodiment of FIG. 11, acompaction scheme that represents values as their relationship to aprincipal value (e.g., a mean value) may appreciably decrease storageoverhead. Here, representation of each the values 1120 as a percentagedeviation from a mean value is a fraction of the storage required torepresent the actual value (e.g., eight bits may be sufficient tocapture DEV[x] using a scaled integer). Other compaction schemes notrepresented in the graphs of FIGS. 7-11 may be employed, such as abitmap index.

Referring now to FIG. 12, a flow diagram illustrates another method 1200for compacting data received from one or more IoT devices, in accordancewith various embodiments of the present disclosure. The method 1200 maybe practiced in one or more computing systems described in the presentdisclosure, such as the data compaction systems 110 and 200 of FIGS.1-2. While the method 1200 illustrates a plurality of sequentialoperations, one of ordinary skill would recognize that one or moreoperations of the method 1200 may be omitted and/or transposed.

The method 1200 may begin with an operation 1205 for receiving, by acomputing system, first data over a network from one or more IoTdevices. At operation 1210, the method 1200 may include identifying anattribute associated with the one or more IoT devices that is common toa first plurality of values included in the first data. Based on thefirst plurality of values, operation 1215 may include determining acompaction scheme that is to indicate the first plurality of values.Accordingly, operation 1220 may include applying the compaction schemeto the first plurality of values to create compaction data. Thereafter,operation 1225 may include storing the compaction data in at least onetable of a database.

In various embodiments, example 1 may be a computing system forprocessing raw data from Internet of Things (IoT) devices, the systemcomprising: network interface circuitry to receive the raw data from theIoT devices over a network; one or more processors; physical memory,coupled with the one or more processors, to store a compaction module tobe loaded into the physical memory for execution by the one or moreprocessors; and the compaction module, coupled with the networkinterface circuitry, to: group values included in the raw data based onat least one attribute associated with at least one IoT device of theIoT devices; select at least one compaction scheme based on the valuesof the group; generate representative data based on application of theselected at least one compaction scheme to the values of the group.Example 2 may be the computing system of example 1, further comprising:a database, wherein the compaction module is to store the representativedata in a table of the database. Example 3 may be the computing systemof example 1, wherein the compaction module is to discard the values ofthe raw data. Example 4 may be the computing system of example 1,wherein at least one of the IoT devices is included in a vehicle, aphone, a medical device, or a meter. Example 5 may be the computingsystem of example 1, wherein the computing system is included in an edgedevice or communicatively coupled with the edge device. Example 6 may bethe computing system of example 1, wherein the compaction module is todiscard the values of the group. Example 7 may be the computing systemof any of examples 1-6, wherein the compaction scheme includes functionfitting, and the compaction module is to fit a function to the values ofthe group. Example 8 may be the computing system of any of examples 1-6,wherein the compaction scheme includes deviation from a representativevalue. Example 9 may be the computing system of example 8, wherein thecompaction module is to generate the representative data as respectivedeviations of the values of the group from the representative value.Example 10 may be the computing system of example 8, wherein thecompaction module is to generate the representative data as anindication of a number of values clustered around the representativevalue within a standard deviation. Example 11 may be the computingsystem of any of examples 1-6, wherein the compaction scheme is based ona bitmap index, and the compaction module is to populate at least onebitmap associated with the bitmap index based on the values of thegroup. Example 12 may be the computing system of any of examples 1-6,wherein the compaction module is to store at least a portion of the rawdata. Example 13 may be the computing system of any of examples 1-6,further comprising: a query processing module, coupled with the networkinterface and the compaction module and to be loaded into the physicalmemory by execution by the one or more processors, to: process a queryreceived by the network interface circuitry; identify responsive databased on the query, wherein the responsive data is to include at leastone of raw data or uncompacted data; and cause the network interfacecircuitry to transmit the responsive data. Example 14 may be thecomputing system of example 13, further comprising: an uncompactionmodule, coupled with the compaction module and the query processingmodule and to be loaded in the physical memory for execution by the oneor more processors, to generate the responsive data based on therepresentative data. Example 15 may be the computing system of example14, wherein the uncompaction module is to generate the responsive datathrough a randomized algorithm for interpolation based on therepresentative data or application of a function indicated by therepresentative data.

In various embodiments, example 16 may be a computer-implemented methodfor compacting data, the method comprising: receiving, by a computingsystem, first data over a network from one or more Internet of Things(IoT) devices; identifying an attribute associated with the one or moreof the IoT devices that is common to a first plurality of valuesincluded in the first data; determining a compaction scheme that is toindicate the first plurality of values; applying the compaction schemeto the first plurality of values to create compaction data; and storingthe compaction data in at least one table of a database. Example 17 maybe the computer-implemented method of example 16, where the applying ofthe compaction scheme comprises: generating a bitmap index based on theattribute and the first plurality of values. Example 18 may be thecomputer-implemented method of example 16, wherein the applying of thecompaction scheme comprises: fitting a function to the first pluralityof values. Example 19 may be the computer-implemented method of any ofexamples 16-18, further comprising: receiving, over the network, arequest for data associated with the one or more IoT devices;determining response data based on the request and the compaction data;and transmitting, in response to the request, the response data over thenetwork. Example 20 may be the computer-implemented method of example19, wherein the determining the response data based on the requestcomprises: generating the response data based on the compaction data.Example 21 may be the computer-implemented method of any of examples16-18, further comprising: determining another compaction scheme that isto indicate a second plurality of values included in the first data,wherein the second plurality of values have the attribute in common withthe first plurality of values; and applying the other compaction schemeto the second plurality of values to create other compaction data.

In various embodiments, example 22 may be one or more non-transitorycomputer system-readable media comprising computing device-executableinstructions, wherein the instructions, in response to execution by acomputing system, cause the computing system to: process raw datareceived over a network from one or more Internet of Things (IoT)devices; group first data from the raw data based on a shared attributeassociated with at least one of the IoT devices; select a compactionscheme based on the first data; generate second data based onapplication of the compaction scheme to the first data; and store thesecond data in a database to indicate the first data. Example 23 may bethe one or more non-transitory computer system-readable media of example22, wherein the instructions further cause the computing system to:process a request received from another computing system; generate thirddata associated with the one or more IoT devices based on the seconddata and the request; and transmit the third data to the other computingsystem in response to the request. Example 24 may include the one ormore non-transitory computer system-readable media of any of examples22-23, wherein the instructions to generate the second data compriseinstructions to: populate at least one bitmap based on the first data,wherein at least one column of the bitmap is associated with the sharedattribute. Example 25 may be the one or more non-transitory computersystem-readable media of any of examples 22-23, wherein the instructionsto generate the second data comprise instructions to: select a meanvalue based on the first data; and generate values of the second databased on values of the first data in relation to the mean value.

In various embodiments, example 26 may be an apparatus comprising: meansfor receiving, by a computing system, first data over a network from oneor more Internet of Things (IoT) devices; means for identifying anattribute associated with the one or more of the IoT devices that iscommon to a first plurality of values included in the first data; meansfor determining a compaction scheme that is to indicate the firstplurality of values; means for applying the compaction scheme to thefirst plurality of values to create compaction data; and means forstoring the compaction data in at least one table of a database. Example27 may be the apparatus of example 26, where the means for applying ofthe compaction scheme comprises: means for generating a bitmap indexbased on the attribute and the first plurality of values. Example 28 maybe the apparatus of example 26, wherein the means for applying of thecompaction scheme comprises: means for fitting a function to the firstplurality of values. Example 29 may be the apparatus of any of examples26-28, further comprising: means for receiving, over the network, arequest for data associated with the one or more IoT devices; means fordetermining response data based on the request and the compaction data;and means for transmitting, in response to the request, the responsedata over the network. Example 30 may be the apparatus of example 29,wherein the means determining the response data based on the requestcomprises: means for generating the response data based on thecompaction data. Example 31 may be the apparatus of any of examples26-28, further comprising: means for determining another compactionscheme that is to indicate a second plurality of values included in thefirst data, wherein the second plurality of values have the attribute incommon with the first plurality of values; and means for applying theother compaction scheme to the second plurality of values to createother compaction data.

Some portions of the preceding detailed description have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the arts. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission, or display devices.

Embodiments of the invention also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer-readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine- (e.g., a computer-)readable storage medium (e.g., read-only memory (“ROM”), random accessmemory (“RAM”), magnetic disk storage media, optical storage media,flash memory devices).

The processes or methods depicted in the preceding figures can beperformed by processing logic that comprises hardware (e.g., circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer-readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed can be performed in a different order. Moreover, someoperations can be performed in parallel rather than sequentially.

Embodiments of the present invention are not described with reference toany particular programming language. It will be appreciated that avariety of programming languages can be used to implement the teachingsof embodiments of the invention as described herein. In the foregoingSpecification, embodiments of the invention have been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications can be made thereto without departing fromthe broader spirit and scope of the invention as set forth in thefollowing claims. The Specification and drawings are, accordingly, to beregarded in an illustrative sense rather than a restrictive sense.

1-20. (canceled)
 21. An edge device, disposed at an edge of a network,for processing raw data from Internet of Things (IoT) devices, thedevice comprising: network interface circuitry to receive, at the edgeof the network, the raw data from the IoT devices; and one or moreprocessors and physical memory, co-located with the network interfacecircuitry, and coupled with network interface circuitry, to store acompaction module to be executed by the one or more processors, wherein:the compaction module, in response to execution, is to cause the edgedevice to: group, at the edge of the network, values included in the rawdata based on at least one attribute of corresponding ones of the IoTdevices from which the raw data is obtained; and generate, at the edgeof the network, representative data based on application of at least onecompaction scheme to the values of the group, wherein the application ofthe at least one compaction scheme is based on metadata that describes:how the at least one compaction scheme is to be applied to the values togenerate the type of representative data.
 22. The edge device of claim21, further comprising: a database co-located with the network interfacecircuitry, the one or more processors and the physical memory, whereinthe compaction module is to store the representative data in a table ofthe database.
 23. The edge device of claim 21, wherein the compactionmodule is to discard the values of the raw data.
 24. The edge device ofclaim 21, wherein at least one of the IoT devices is included in avehicle.
 25. The edge device of claim 21, wherein the compaction moduleis to further select the at least one compaction scheme based on thevalues of the group.
 26. The edge device of claim 21, wherein thecompaction module is to discard the values of the group.
 27. The edgedevice of claim 21, wherein, when the metadata indicates therepresentative data to be generated is a function, the compaction schemeincludes a function fitting compression scheme or a piecewise compactionscheme, and wherein the compaction module is to: fit a function to thevalues of the group, generate the function as a linear function or aquadratic function when the compaction scheme includes the functionfitting compression scheme, and generate the function as a piecewiselinear function when the compaction scheme includes the piecewisecompaction scheme.
 28. The edge device of claim 21, wherein, when themetadata indicates the representative data to be generated is theprincipal value, the compaction scheme includes deviation from theprincipal value.
 29. The edge device of claim 28, wherein the compactionmodule is to generate the representative data as respective percentagedeviations of the values of the group from the principal value or asrespective standard deviations of the values of the group from theprincipal value.
 30. The edge device of claim 28, wherein the compactionmodule is to generate the representative data as an indication of anumber of values clustered around the representative value within astandard deviation.
 31. The edge device of claim 21, wherein, when themetadata indicates the representative data to be generated is a bitmapindex, the compaction module is to populate at least one bitmapassociated with the bitmap index based on the values of the group. 32.The edge device of claim 21, wherein the compaction module is to storeat least a portion of the raw data.
 33. The edge device of claim 21,wherein the physical memory further stores a query processing module, tobe executed by the one or more processors, to cause the edge device to:process, at the edge of the network, a query received, from the network,by the network interface circuitry; identify, at the edge of thenetwork, responsive data based on the query, wherein the responsive datais to include at least one of raw data or uncompacted data; and causethe network interface circuitry to transmit the responsive data furtherinto the network.
 34. The edge device of claim 33, wherein the physicalmemory further stores an uncompaction module, coupled with thecompaction module and the query processing module, to be executed by theone or more processors, to generate the responsive data based on therepresentative data.
 35. The edge device of claim 34, wherein theuncompaction module is to generate the responsive data through arandomized algorithm for interpolation based on the representative dataor application of a function indicated by the representative data. 36.The edge device of claim 21, wherein the edge device is a selected oneof a routers, a switch, an integrated access devices (IADs), or amultiplexer.
 37. The edge device of claim 21, wherein, when therepresentative data is a bitmap index, and the compaction scheme is aword-aligned hybrid bitmap compression scheme.
 38. The edge device ofclaim 21, wherein the representative data includes an indication of anumber of values comprising the raw data to be represented by therepresentative data.
 39. The edge device of claim 21, wherein thecompaction scheme is explicitly indicated or specified by a set ofparameterized rules.
 40. The edge device of claim 21, wherein themetadata further describes how the representative data is to be stored.41. A computer-implemented method for compacting data, the methodcomprising: receiving, by an edge device located at an edge of anetwork, first data from one or more Internet of Things (IoT) devicesintegrated with a vehicle; identifying, by the edge device, at the edgeof the network, an attribute associated with the one or more of the IoTdevices integrated with the vehicle that is common to a first pluralityof values included in the first data; determining, by the edge device,at the edge of the network, a compaction scheme that is to indicate thefirst plurality of values; applying, by the edge device, at the edge ofthe network, the compaction scheme to the first plurality of values tocreate compaction data for the first data forwarding onto the network,by the edge device, from the edge of the network, the compaction datacreated for the first data.
 42. The method of claim 41, wherein theapplication of the determined compaction scheme is based on metadatathat describes: a type of compaction data to be generated, wherein thetype of compaction data to be generated comprises a function, a bitmapindex, or a principal value, how the selected at least one compactionscheme is to be applied to the values to generate the type of compactiondata, and how the compaction data is to be stored.
 43. One or morenon-transitory computer system-readable media comprising computingdevice-executable instructions, wherein the instructions, in response toexecution by an edge device, located at an edge of a network, cause theedge device to: process, at the edge of the network, raw data receivedfrom one or more Internet of Things (IoT) devices; group, at the edge ofthe network, first data from the raw data based on a shared attributeassociated with at least one of the IoT devices; select, at the edge ofthe network, a compaction scheme based on the first data; generate, atthe edge of the network, second data based on application of thecompaction scheme to the first data, wherein the application of theselected compaction scheme is based on metadata that describes: how thesecond data is to be stored; and store the second data in a database toindicate the first data.
 44. The one or more non-transitory computersystem-readable media of claim 43, wherein the application of theselected compaction scheme is further based on the metadata furtherdescribing the type of second data to be generated, and how the selectedcompaction scheme is to be applied to the first data to generate thetype of second data.
 45. The one or more non-transitory computersystem-readable media of claim 44, wherein the instructions furthercause the edge device to: process, at the edge of the network, a requestreceived from the network; generate, at the edge of the network, thirddata associated with the one or more IoT devices based on the seconddata and the request; and transmit, at the edge of the network, thethird data into the network to respond the request.
 46. The one or morenon-transitory computer system-readable media of claim 43, wherein oneor more of the IoT devices are integrated with a vehicle.